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EXPERIMENT MANAGEMENT SYSTEM, METHOD AND MEDIUM 

Inventors: Badri N. Krishnamurthy, Parris CM. Hawkins 
BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention concerns computer-related methods systems and mediums for 
managing experiments. More specifically, it relates to managing experiments concerning 
changes in a process, for example processes for controlling semiconductor technology 
manufacture. 

Related Art 

Machines, materials and processes in most industries are becoming increasingly complex 
and costly. Meanwhile, a need has arisen for the continuing improvement of processes and of 
machine and material quality. 

Semiconductors and other products are typically manufactured under control of pre- 
defined processes. These pre-defined processes may be highly complex. For example, a pre- 
defined manufacturing process for producing semiconductor chips might contain five hundred to 
seven hundred and fifty steps. Moreover, each of these steps might have several variables, for 
example six variables, that are significant. 

In order to improve manufacturing or test theories, it is often desirable to perform 
experiments by changing some small portion of the base manufacturing process. For example, 
an engineer might want to make one of the layers on a semiconductor ten percent thicker. This 
might entail performing the recipe for that step for an extra 15 seconds, with perhaps some 
adjustments in subsequent steps. Typically the engineer does not create a new base process 
including the modifications to adapt to the desired test, since that would be too time consuming. 
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Unfortunately, such an experiment using conventional techniques requires manual 
intervention and manual tracking of results. Accordingly, the engineer or operator performing 
the experiment would obtain a number of semiconductor chips and process them outside of an 
automated (e.g., production or mock-production manufacturing) environment. Thus, the 
products on which the experiment is performed need to be removed from the automated 
environment, which is both time-consuming and allows for the potential introduction of 
extraneous factors which may ultimately (and inadvertently) affect the results of the experiment. 
In addition, such removal of the semiconductor chips makes it difficult to coordinate manual 
tracking of changes or experiment history, and to control experiments and to analyze overall 
results. 

Consequently, for research and development engineers, operators and other users 
working in factory settings, there remains a need for experiments on changes to existing 
processes to be flexible, easy and traceable. 

SUMMARY OF THE INVENTION 

The present invention alleviates the problems of the conventional techniques described 
above by providing systems, methods and mediums for automating experiments within an 
automated (e.g., production or mock-production manufacturing) environment without the need to 
disassociate the test subject (e.g., the semiconductor chip or chips) from that environment. An 
"experiment," according to at least some embodiments of the present invention, is a pre-planned 
deviation of at least some portion of an established (e.g., pre-defined) process utilizing the 
automated environment. 

According to at least some embodiments of the present invention, experimentation begins 
with an experiment order (i.e., request to initiate an experiment), which is first originated as an 
informal request, submitted to a computerized system, routed through various defined users, 
perhaps modified, and ultimately approved. In facilitating the implementation of the requested 
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experiment, experiment management includes four conceptually distinct stages: order 
management, setup, execution, and analysis. The order management component of the invention 
assists in automatically navigating the formalization of the experiment order (mentioned above) 
and tracking the experiment. The setup stage typically handles the manual or automated 
5 translation of the experiment from the generalized statements, requirements, or proposed results 
into data defining a specific process ready to execute by the automated environment. The 
execution stage includes the execution of the experiment itself via the automated environment 
based on the process data, including the collection of experiment results. In the analysis stage, 
results of the experiment are reported and analyzed. 
10 In accordance with at least some embodiments of the present invention, in operation, an 

;* ~~ : 

□ experiment order is received, the experiment order including at least some deviation from a base 

c 

s y process capable of operating in an automated environment. An approval of the experiment order 
is then obtained. At least a portion of the experiment order is translated into processing data 
s » suitable for implementation by said automated environment, and stored. The experiment is 



: automated environment according to the processing data. 

in 

;!;? Further, the invention may include storing data defining the experiment order, 

I 1 * distributing the experiment order to a plurality of users, obtaining changes to the experiment 

order from at least one of the users, and receiving the approval for the experiment order from at 
20 least one user. Moreover, documents may be attached to the experiment request. 

Additionally, information indicating a state change of the experiment request may be 

published, responsive to a document attached to the experiment request or to a change in state of 

the experiment order. 



25 production product (i.e., a control, which could be, e.g., a product which was processed before or 
after the test product, and which was processed according to the base process); the processing 




caused to be executed in conjunction with at least some portion of said base process via the 



Moreover, the experiment may produce at least one test product and at least one 
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data may include an indication of the base process, the changes to the base process, and a split- 
off of a control set (i.e., the products subject to the experiment); and the split-off of a control set 
may produce the at least one production product according to the base process and the changes to 
the base process may produce the at least one test product. The results of the execution of the 
5 experiment may be stored. 

BRIEF DESCRIPTION OF THE FIGURES 
The above mentioned and other advantages and features of the present invention will 
become more readily apparent from the following detailed description in the accompanying 

10 drawings, in which: 

Q 

u 3 Figure 1 is a block diagram of a computerized process control system which may be used 

: i J 

j s y in connection with at least some embodiments of the present invention. 

CS 

'Jj* Figure 2 is a flow chart of an overall process for experiment management according to at 

_f least some embodiments of the invention. 

;Il5 Figures 3 A and B are a flow chart of an order management process portion of the overall 

process of Figure 2. 

;r Figure 4 is a flow chart of a setup process portion of the overall process of Figure 2. 

i sa Figure 5 is a flow chart of an execution process portion of the overall process of Figure 2. 

Figure 6 is a flow chart of an analysis process portion of the overall process of Figure 2. 
20 Figure 7 is a diagram illustrating definition of an experiment. 

Figure 8 is an exemplary user interface for an experiment editor, used in connection with 
at least some embodiments of the present invention. 

Figure 9 is an exemplary user interface for the experiment editor, illustrating attachments, 
used in connection with the invention. 
25 Figure 10 is an exemplary user interface for an experiment editor, illustrating experiment 

content, used in connection with at least some embodiments of the present invention. 
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Figure 1 1 is an exemplary user interface for an experiment editor, illustrating wafer level 
split details, used in connection with at least some embodiments of the present invention. 
Figure 12 is an illustration of at least some embodiments of an experiment. 



5 DETAILED DESCRIPTION 

The following detailed description includes many specific details. The inclusion of such 
details is for the purpose of illustration only and should not be understood to limit the invention. 
Throughout this discussion, similar elements are referred to by similar numbers in the various 
figures for ease of reference. 
10 As indicated above in the Summary section, an "experiment," according to at least some 

□ embodiments of the present invention, is a pre-planned deviation of at least some portion of a 

Q 

ry base process utilizing an automated environment. Typically an experiment is performed on 

C£J 

materials, such as semiconductor chips, that are produced as a result of the automated process. 

i, 3 

» Also as indicated above, at least some embodiments of the present invention envision that 

|i5 experiment management includes four conceptually distinct stages: order management, setup, 

: Si=r 

I'^f execution, and analysis. Although these stages are conceptually distinct, they may temporally 
overlap. 

U 

According to at least some embodiments of the present invention, reports, memos, forms, 
files, and other documents may be associated with a particular experiment throughout the order 

20 management and setup stages. These may be reviewed by users allowed access to the 
experiment. This permits users and reviewers to comment on the experiment, provide 
background information, provide appropriate forms, attach relevant information, etc., in a user- 
friendly, highly flexible fashion. Due to its flexibility, it invites users to provide input and 
should result in higher quality experiments. 

25 Reference is now made to Figure 1, a block diagram generally illustrating a computerized 

process control system which may be used in connection with at least some embodiments of the 
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present invention. As is illustrated, the experiment order 101 is input to a computerized system, 
referred to generally as a controller 103. The experiment order 101 contains a description, such 
as in text, of a desired experiment. The experiment order 101 could be, for example, a word 
processing document containing text. As one alternative, it could be input from a menu. The 
5 experiment described in the experiment order 101 is a deviation from an -existing automated 
process for creating a product, although it is not necessarily described in the order as a deviation 
from a particular process. 

The controller 103 has access to various stored processes 111, such as manufacturing 
processes lor semiconductor chips . The controller 103 could be a general purpose computer, or a 
10 special purpose computer specially programmed, or other automated system or distributed 
i,3 system. (In general, such computers as used here, or whose use may be apparent from the 
ry context of the discussion, can be any number of different types of computers, including those 
:5 2 containing processors from Intel Corporation of Santa Clara, CA, wherein these computers can 
„« contain any number and different types of storage devices serving as computer-readable 
i«k5 mediums; in addition, it is contemplated by at least some embodiments of the present invention 
^ that the computer-readable medium be a transmission). The stored processes 111 comprise a 
^ number of automated steps in a manufacturing process. The actual format of the contents of 
f"* these steps is defined by the system and devices in the system. Some of the steps in the 
processes utilize recipes, stored in a recipe database 113. Recipes may be shared by various 
20 processes. The controller 103 controls the processing of an automated environment such as 
production system 105, which ultimately produces production products 107, or following an 
experiment, produces test products 109. The invention thereby allows users to submit 
experiment requests, create derivations of base processes, and to track the status of experiment 
requests. 

25 Reference is made to Figure 2, a flow chart of an overall process for experiment 

management according to at least some embodiments of the present invention. The four 
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conceptual stages (as mentioned above) included: order management 201, Manufacturing 
Execution System (MES) setup 203, execution 205, and analysis 207. 

At the order management stage 201, further defined below, the experiment order is 
defined. Typically, an experiment would be defined in the experiment order as a set of 
requirements, and may be specified as a deviation from an existing process. The experiment 
order is subject to routing, review, and change by various personnel, prior to being approved for 
the next stage. 

At the MES setup stage 203, the experiment order is translated into the experiment setup, 
that is, specific processing data which can be executed by components in the production system. 
The processing data is in a format which is expected by the production system components. In 
typical situations, data to execute the experiment is interjected between (and/or replaces existing) 
steps of a base process. 

At the execution stage 205, the execution of materials is performed, based on the 
experiment setup. Most or all of this stage is performed automatically by the production system 
components. The results of each step in the setup implemented at this execution stage 203 are 
recorded. 

At the analysis stage 207, the results of the experiment are reported and analyzed. This 
may be done automatically by a computer, and/or may include analysis by the user. 

Reference is made to Figures 3A and 3B, a flow chart of an example order management 
stage 201 of the overall process of Figure 2, as envisioned by at least some embodiments of the 
present invention. This stage allows the experiment to be requested and be performed following 
experiment request review and sign-off. At step 301, the experiment is initially defined by a 
requestor. In order to facilitate experiments, it is envisioned that requests can be submitted in 
any appropriate form. One appropriate form is a textual description in an electronic document. 
Note that the experiment may be informally described. It is not necessary for the initial 
experiment request to define the experiment as a variation from an existing process. 
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At step 303, the experiment object (or other storage for experiment data) is created. 
Initial information is collected to identify the requestor and the experiment. The information is 
stored, such as in an object. The experiment request is then distributed to appropriate users 
identified in a distribution list. 
5 At step 305, a user who received the experiment request (e.g., for review) may attach 

external files, memos, forms, or other documents to the experiment request. The ability to 
associate documents with the experiment request can be used to facilitate user interaction 
concerning the experiment request. These documents may then be reviewed by other users. 
At step 307, the user (or automated entity) determines the changes to be made to a particular base 

10 process. The user (or automated entity) may also determine the base process which is to be 

□ 

j. 3 modified. Also, at step 309, the user (or automated entity) will determine when to split off a lot 

ry from the control set, and the lot-specific transactions that are to be made. At step 311, the user 

^ (or automated entity) determines what recipe changes, if any, need to be made. Having 

» determined the specified changes to be made to the base process, the system receives and stores 

'-15 the changes as processing data. At step 313, the experiment, as it has been tweaked by the users, 

hex 

; ,y is sent for sign-off, described in Figure 3B. At step 315, if the experiment has been approved by 

E3 

j;~ the users, the process ends 317 and the experiment proceeds to the next conceptual stage. 
Otherwise, the process returns to step 305 for further handling. 

Figure 3B illustrates one embodiment of the sign-off process. At step 321, a user who 

20 received the experiment request (e.g., for review) may attach external files, memos, forms, or 
other documents to the experiment request, which may then be reviewed by other users. At step 
323, if documents are attached or deleted to the experiment request, or at step 325 if there was a 
state change for the experiment request, such information is published 327. One appropriate 
method for publication is to send such information to listed users via e-mail. A state change 

25 would include, for example, a "sign-off on the experiment (or portion thereof). At step 329, if 
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an indication of final approval (or affirmative lack of approval) has not been received, the 
process repeats at step 321. If final approval has been received, the stage is ended 331. 

Reference is made to Figure 4, a flow chart of a setup stage 203 portion of the overall 
process of Figure 2. During the setup stage, a user can set up the particular experiment. For 
- 5 example, a user could set up experiment-specific data, for example a reticle or recipe details. At 
step 401, a user (or automated entity) retrieves and reviews the experiment order. As indicated 
above, the experiment order may be an informal description of an experiment. A user can 
determine how a process should be implemented to effect the requested experiment, or the 
process can be automated, for example, by parsing the description of the experiment and 
10 identifying certain key words or phrases that are indicative of what is requested. At least some 
ijj embodiments of the present invention envision that this can be done utilizing, e.g., various expert 
ry system techniques. At least some embodiments of the present invention also envision some 

i:o 

>s » combination of automation and user participation. 

J Still referring to Figure 4, at step 403, the user (or automated entity) determines the 

%5 changes to be made to a particular base process. The user (or automated entity) may also 

! a=? 

' * determine the base process which is to be modified. Also, at step 405, the user (or automated 

1 

l \* entity) will determine when to split off a lot from the control set, and the lot-specific transactions 

i 3 ~ that are to be made. At step 407, the user (or automated entity) determines what recipe changes, 
if any, need to be made. Having determined the specified changes to be made to the base 

20 process, the system receives and stores the changes as processing data. 

Reference is made to Figure 5, a flow chart of an execution stage 205 of the overall 
process of Figure 2. At this point, the experiment has been defined in processing data which can 
be input to the automated environment. The experiment can then be processed in a manner 
which is transparent to the automated environment. At step 501, the automated environment 

25 receives the processing data for the modified process. At step 503, the automated environment 
executes a step of the processing data. If there are any test results to be stored, at steps 505-507, 
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the system stores the test results. At step 509, if processing is not complete, the automated 
environment returns to continue processing at step 503. When processing is complete, this stage 
ends at step 511. 

Reference is made to Figure 6, a flow chart of an analysis stage 207 of the overall process 
5 of Figure 2. Experiment history setup information and history data is available for use in 
analysis and reporting. The experiment results are collected at step 601. At step 603, the 
experiment results are made available for any analysis. For example, a user may wish to make a 
manual analysis of the results. At step 605, the automated environment performs any requested 
computerized analysis. If there are any proposed changes to the experiment, at steps 607-609, 

10 the user may generate another experiment request. The analysis is completed at step 611. 

Q 

ij3 Reference is made to Figure 7, a diagram illustrating the defining of an experiment, as 

h3 

ry contemplated by at least some embodiments of the present invention. Specifically, the 
experiment 701 initially is associated with stored data including attribute information 703, for 
r « example defined by the user, and operation information 705, defining how the experiment 
j!l5 operates. An experiment initially may be created from scratch, or may be copied from another 
j'? experiment used as a template. Typical attributes would include sufficient information to 
identify useful information about the experiment, such as an experiment identifier, an experiment 

Q 

objective, a requestor name, an experiment name, a requestor e-mail address. 

When the experiment is initially defined, a starting state will be "underchange" 707 

20 (indicating that the experiment may be changed), and once the experiment is approved, the 
ending stage is effective (distributed) 711. There may be a series of user-defined states 709 
which are under control of the user, subsequent to the underchange state, and prior to the 
effective state. The effective state is entered after the experiment is approved and signoff is 
obtained. Preferably, a user cannot change the contents of an experiment without appropriate 

25 permission. There may be other user-defined attributes, as well as attached external documents 
and/or files, and a user-defined state model. According to one possible implementation, the 
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experiment is implemented as an object. Note that this state table corresponds to the order 
management process portion. 

Figures 8-11 are examples of a potential user interface to be used in connection with at 
least some embodiments of the present invention. First, reference is made to Figure 8, one 
aspect of an exemplary user interface for an experiment editor. Here, the user may provide 
information about the experiment 811, about experiment attributes 813, and optionally about 
experiment category 815. Experiment information may include an objective 801, which may 
summarize a description of the experiment. Other experiment information includes requestor 
identification information 803 (for example, name, e-mail address); the basic process or state 
model 805 for the experiment; and optionally an effective date 807 after which the experiment 
request will expire. The information collected in this initial interface is associated with the 
experiment request. 

Reference is made to Figure 9, another aspect of an exemplary user interface for the 
experiment editor, illustrating attachments used in connection with at least some embodiments of 
the present invention. In such embodiments, documents such as files, memos, forms, web 
addresses, etc., without limitation, may be attached to or otherwise associated with the 
experiment request. Figure 9 lists, by way of example, several documents, by file name 909, 
which are attached to the experiment request: a local document experiment.doc 901; a filepath 
for another document C:\Experiment\Experiment.doc 903; a web site www.consolium.com 905; 
and an http document http://www.consilium.com/corp_events.html?phase=ge 907. The user 
interface of the present example also indicates whether or not the file is simply a reference 911. 

Reference is made to Figure 10, another aspect of an exemplary user interface for the 
experiment editor, illustrating experiment content, used in connection with at least some 
embodiments of the present invention. This exemplary user interface allows access to 
experiment content 1001, physical split details 1003, and merge details 1005, the split treating 
the standard and test materials differently, and the merge detailing how the standard and test 
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materials are treated when merged after the split. The experiment content 1001 provides the file 
controlling the experiment process. Here, it names the experiment process 1007, the experiment 
route 1009, and the experiment operation 1011. Note that additional information on the 
experiment may be provided, such as whether the processing is pre- or post-split 1013. 

Reference is made to Figure 11, an exemplary user interface for the experiment editor, 
illustrating wafer level split details, used in connection with the invention. Here, the processing 
data provides specifics, at lot level, slot level, or unit level 1 101. The present example concerns 
a slot level split. As is illustrated, the split details 1103 provide the slots and the quantity to be 
split; as well as the process plans 1 105 to be associated with each split. 

Reference is made to Figure 12, illustrating at least some embodiments of an experiment 
as contemplated by the present invention. Each experiment order 1201 may have associated with 
it various documents, such as files 1203, forms 1205, memos 1207, and experiment results 1209. 
Users can add or delete the document to/from the experiment order. Preferably, an attachment of 
a document will be considered an event, and may result in the publication of the event for 
example by e-mail or Workflows. 

An experiment order may be copied by a user, together with attached documents, 
attributes, and other correlated information 

Also, according to at least some embodiments of the present invention, changes to the 
experiment order are stored in a history. Stored changes could include changes to native 
attributes, external document additions/deletions, and associated with other objects. 

Consider an example of an experiment, with reference to Figures 3 through 6. In this 
example, the user wants a specified layer of a chip to be 10 % thicker. The experiment in this 
example is an idea from an engineer. The experiment request is defined by a user, and submitted 
to the system at step 301 through 303. It could be a very general request with a simple textual 
description. An experiment object is created for the experiment request, and the experiment 
request is routed to the appropriate users for approval, at steps 305 through 313. The approval 
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may be automated, such as delivery via e-mail awaiting a marking as approved. As shown in 
steps 321 through 329, until sign-off is received for the experiment, users may attach and/or 
delete relevant files, memos, etc. to the experiment object. If there are attachments or deletions, 
or if the experiment has changed state, the event is published to the users, shown in steps 323 
through 327. The review process continues until sign-off is received. 

Once sign-off is received, the experiment order is reviewed and translated to processing 
data, as shown in Figure 4. This review and translation may be a manual process done by a 
person with the appropriate experience. In addition, it may also be performed (in whole or part) 
by automated means. In any event, it could be determined at step 403, for example, that wafers 1 
- 11 in the lot will be the control (i.e., the established steps will not be effected), and the 
remainder of the wafers in the lot will be the test product. Also, it could be determined that a 
particular parameter in the 500 th cycle of a standard base process must be changed from 100 to 
200. It would be specified at step 405 that the controls will be split off from the other 
processing. If it was necessary, a new recipe would be created or an existing recipe would be 
modified at step 407. All of the wafers will be under automated control. The two lots will be re- 
united and held or delivered for analysis. The information related to the variations from the base 
process, specific execution transactions, and any recipe change are stored as processing data. 
Note that the experiment could call for additional or different information to be collected as part 
of the processing results. 

The experiment is then run, as shown in Figure 5. At this point, the experiment 
processing data are handled no differently from a regular control job. That is, no exception 
processing is required. The processing data is input into the manufacturing system at step 501, 
and the test proceeds automatically by executing the processing data at step 503. Test results 
that are generated during execution of the experiment are stored at steps 505-507. 

Following the experiment, test product might be reclassified from test materials to 
standard production materials, if within tolerances, and shipped to customers. Alternatively, the 
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non-standard processed materials could be scrapped, or saved for further analysis, as shown in 
Figure 6. 

While this invention has been described in conjunction with the specific embodiments 
outlined above, many alternatives, modifications and variations will be apparent to those skilled 
in the art. Accordingly, the preferred embodiments of the invention as set forth are intended to 
be illustrative and not limiting. Various changes may be made without departing from the spirit 
and scope of the invention as defined in the following claims. 

For example, it would be possible to define an entire experiment from scratch. A typical 
semiconductor manufacturing process is 500 to 750 steps, so it may often be more efficient to 
define an experiment as a variation from an existing process. 

As another example, the controller may be a general purpose computer, a specially 
programmed special purpose computer; it may also be implemented as a distributed computer 
system rather than as a single computer. 
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